Better Picture For Inexpensive Cameras

ABSTRACT

A method for correcting a radial distortion in an image includes receiving a radial distortion parameter from the user and calculating new intensity values of points in the image to correct a radial exposure distortion in the image.

CROSS REFERENCE TO RELATED APPLICATION

This application is a division of U.S. patent application Ser. No.10/641,679, filed on Aug. 15, 2003, and incorporated herein byreference.

FIELD OF INVENTION

This invention relates to image enhancing software for low-cost cameras.

DESCRIPTION OF RELATED ART

The pictures captured by low-cost cameras, such as web cams and camerasembedded in cellular phones, are generally of poor quality. One of thereasons for the poor quality is the use of low-cost optical systemsutilizing a fixed or almost fixed lens having a very short focal length.Another reasons for the poor picture quality is the use of low-cost CMOS(complementary metal oxide semiconductors) imaging sensors instead ofCCD (charge-coupled device) sensors.

A low-cost camera may produce a picture with a nonlinear (barrel)distortion as shown in FIG. 1A. Thus, what is needed is a method tocorrect the nonlinear distortion in the pictures captured by low-costcameras.

A low-cost camera may also produce a picture with an uneven radialexposure where the center portion of a picture is brighter than thesurrounding portions as shown in FIG. 4A. Thus, what is needed is amethod to correct the uneven radial exposure in the pictures captured bylow-cost cameras.

A low-cost camera may further produce a picture with noise, andespecially poor exposure, color cast, poor contrast, poor brightness,and poor focus as shown in FIGS. 6A, 7A, and 8A. Specifically, FIG. 6Ashows one example of both poor exposure and color cast. FIG. 7A showsone example of poor contrast. FIG. 8A shows one example of poorbrightness. Thus, what is needed is a method to reduce the noise,enhance the contrast, correct the color cast, and improve the focus ofthe pictures captured by low-cost cameras.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1A is an image with nonlinear distortion in one embodiment of theinvention.

FIG. 1B is the image of FIG. 1A after the nonlinear distortion has beencorrected in one embodiment of the invention.

FIG. 2A is a flowchart of a method for correcting the nonlineardistortion in one embodiment of the invention.

FIG. 2B is a calibration template used by the method of FIG. 2A in oneembodiment of the invention.

FIG. 2C is an image of the calibration template with nonlineardistortion in one embodiment of the invention.

FIG. 3 is a flowchart of another method for correcting the nonlineardistortion in one embodiment of the invention.

FIG. 4A is an image with radial exposure in one embodiment of theinvention.

FIG. 4B is the image of FIG. 4A after the radial exposure has beencorrected in one embodiment of the invention.

FIG. 5 is a flowchart of a method for correcting radial exposure in oneembodiment of the invention.

FIG. 6A is an image that has poor exposure and color cast in oneembodiment of the invention.

FIG. 6B is the image of the FIG. 6A after the image has been enhanced inone embodiment of the invention.

FIG. 7A is an image that has poor contrast in one embodiment of theinvention.

FIG. 7B is the image of the FIG. 7A after the image has been enhanced inone embodiment of the invention.

FIG. 8A is an image that has poor brightness in one embodiment of theinvention.

FIG. 8B is the image of the FIG. 8A after the image has been enhanced inone embodiment of the invention.

FIG. 9 is a flowchart of a method for enhancing an image in oneembodiment of the invention.

SUMMARY

In one embodiment of the invention, a method for correcting a radialdistortion in an image includes receiving a radial distortion parameterfrom the user and calculating new intensity values of points in theimage to correct a radial exposure distortion in the image.

DETAILED DESCRIPTION Optical Distortion Correction

In one embodiment of the invention, an optical nonlinear distortionmodel of a camera is assumed to comply with the following equations:x′=x·[1+k(x ² +y ²)]; and  (1.1)y′=y·[1+k(x ² +y ²)].  (1.2)In Equations 1.1 and 1.2, (x,y) are the coordinates of a point (e.g., apixel) with the nonlinear distortion (i.e., before correction), (x′,y′)are the coordinates of the point without the nonlinear distortion (i.e.,after correction), and k is a global distortion parameter for the entireimage. In this model, the origin of the coordinates is at the center ofthe image. Global distortion parameter k may change when the camerafocus changes.

In the above optical model, a straight line equation in polarcoordinates without nonlinear distortion is as follows:ρ′ cos(θ′−α)=R.  (2.1)(ρ′,θ′) are the polar coordinates of any point (e.g., pixel) on astraight line without nonlinear distortion (i.e., after correction), and(α,R) are the parameters of the straight line. With the nonlineardistortion, the polar coordinates of any point (ρ′,θ′) are calculated asfollows:ρ′=ρ(1+kρ ²); and  (2.2)θ′=θ.  (2.3)In Equations 2.2 and 2.3, (ρ,θ) are the polar coordinates of a pointwith the nonlinear distortion (i.e., before correction), (ρ′,θ′) are thepolar coordinates of the point on a straight line without nonlineardistortion (i.e., after correction), and k is the global distortionparameter. Thus, the straight line equation in polar coordinates withthe nonlinear distortion becomes:ρ(1+kρ ²)cos(θ−α)=R.  (2.4)Equation 2.4 has three variables of parameters (α,R) and globaldistortion parameter k, which can be solved with the polar coordinatesof three points known to lie in a straight line before the nonlineardistortion.

FIG. 2A illustrates a method 10 for software to correct the nonlineardistortion in an image in on embodiment of the invention. Method 10 usesa calibration template to automatically detect and correct the nonlineardistortion. Method 10 is well suited for cameras with a fixed focusotherwise the software may need to be recalibrated each time the focusis changed. The software implementing method 10 can be located on thecamera to process the image in real time or on a computer to process theimage offline.

In step 12, a camera captures an image of a calibration template 22.FIG. 2B shows a calibration template 22. FIG. 2C shows calibrationtemplate 22 with nonlinear distortion as captured by the camera.Calibration template 22 includes L-shaped calibration patterns 24-1,24-2, 24-2, 24-3, 24-4, 24-5, 24-6, 24-7, and 24-8 (collectively“calibration patterns 24”). Referring to FIG. 2B, calibration patterns24 in the same column have their outer corners aligned vertically whilecalibration patterns 24 in the same row have their outer corners alignedhorizontally.

In step 14, the software detects at least (1) a first set of three ormore corners known to be on a first straight line, and (2) a second setof three or more corners known to be on a second straight line. Forexample, the software detects corners 25 and 26 of pattern 24-1, corners27 and 28 of pattern 24-2, corners 29 and 30 of calibration pattern 24-3known to lie on a straight line 33 if the camera does not have nonlineardistortion. Similarly, the software detects corners 32 and 33 of pattern24-1, corners 34 and 35 of pattern 24-8, and corners 36 and 37 ofpattern 24-7 known to lie on a straight line 38 if the camera does nothave nonlinear distortion.

In step 16, the software determines distortion parameter k of equations2.4 from the coordinates of the detected corners. Specifically, thesoftware first determines the polar coordinates of corners 25 to 30 and32 to 37. The software then solves parameters (α,R) and distortionparameter k in equation 2.4 for each set of the corners known to be onthe same straight line without the nonlinear distortion. For example,the software first determines parameters (α₁, R₁) and k₁ from corners 25to 30 known to be on straight line 31 using the following formula:ρ(1+k ₁ρ²)cos(θ−α₁)=R ₁.  (2.5)In Equation 2.5, (ρ,θ) are polar coordinates of a point on line 31 withthe nonlinear distortion, k₁ is the local distortion parameter for line31, and (α₁, R₁) are parameters of the line 31.

The software then determines parameters (α₂, R₂) and k₂ with corners 32to 37 known to be on straight line 38 with the following formula:ρ(1+k ₂ρ²)cos(θ−α₂)=R ₂.  (2.6)In Equation 2.6, (ρ,θ) are polar coordinates of a point on line 38 withthe nonlinear distortion, k₂ is the local distortion parameter for line38, and (α₂, R₂) are parameters of line 38.

In each set there are three unknowns and therefore 3 points arenecessary to solve for these unknowns. Thus, in each set only the 3corners that are farthest from each other are selected.

The software then choose one of local distortion parameters k₁ and k₂that best fits the equations of all points in the image as the globaldistortion parameter k. Specifically, the software uses k₁ instead of k₂to fit corners 32 to 37 on line 38. The software thus varies the valueof (α₂, R₂) to achieve the smallest error as follows: $\begin{matrix}{e_{1} = {\sum\limits_{i = 1}^{3}\quad{{{{{\rho_{i}( {1 + {k_{1}\rho_{i}^{2}}} )}{\cos( {\theta_{i} - \alpha_{2}} )}} - R_{2}}}^{2}.}}} & (2.7)\end{matrix}$In Equation 2.7, e₁ is the error achieved using k₁ instead of k₂ to fitcorners 32 to 37 on line 38 and (ρ_(i), θ_(i)) are the polar coordinatesof the three farthest corners on line 38. In one embodiment, theLevenberg-Marquardt method is used to minimize error e₁.

Similarly, the software uses k₂ instead of k₁ to fit corners 25 to 30 online 31. Again, the software varies the value of (α₁,R₁) to achieve thesmallest error as follows: $\begin{matrix}{e_{2} = {\sum\limits_{i = 1}^{3}\quad{{{{{\rho_{i}( {1 + {k_{2}\rho_{i}^{2}}} )}{\cos( {\theta_{i} - \alpha_{1}} )}} - R_{1}}}^{2}.}}} & (2.8)\end{matrix}$In Equation 2.8, e₂ is the error achieved using k₂ instead of k₁ to fitcorners 25 to 30 on line 31 and (ρ_(i),θ_(i)) are the polar coordinatesof the three farthest corners on line 31. In one embodiment, theLevenberg-Marquardt method is used to minimize error e₂.

If e₁ is less than e₂, then the software selects k₁ as global distortionparameter k. Conversely, if e₂ is less than e₁, then the softwareselects k₂ as global distortion parameter k.

In step 18, the software determines if the focus of the camera haschanged. If so, then step 18 is followed by step 12 and globaldistortion parameter k for the camera is recalibrated. Otherwise step 18is followed by step 20.

In step 20, the software corrects nonlinear distortions from imagescaptured with the camera. Specifically, the software uses globaldistortion parameter k and equations 1.1 and 1.2 to calculate the newcoordinates of each point in the images. FIG. 1B shows one exemplaryresult.

FIG. 3 illustrates a method 40 for software to correct the nonlineardistortion in an image in on embodiment of the invention. Method 40 usesuser inputs to correct the nonlinear distortion. Compared to method 10,method 40 is well suited for cameras with auto focus.

In step 42, the software receives from a user the identity of threepoints (e.g., pixels) in the image that are known to be on the samestraight line.

In step 44, the software determines distortion parameter k usingequations 2.4 from the coordinates of the three points as describedabove in step 16.

In step 46, the software corrects the nonlinear distortion from theimage as described above in step 20.

Radial Exposure Correction

In one embodiment of the invention, a radial exposure model of a camerais assumed to comply with the following equations:I′(x,y)=I(x,y)^(1+k(x) ² ^(+y) ²⁾ .  (3.1)

In Equation 3.1, I′(x, y) is the intensity value of a point (x,y) in theimage with radial exposure, I(x,y) is the intensity value of the point(x,y) in an ideal image without radial exposure, (x,y) are thecoordinates of any point (e.g., pixel) in the image, and k is adistortion parameter. In this model, the origin of the coordinates is atthe center of an image.

FIG. 5 illustrates a method 50 for software to correct radial exposurein one embodiment of the invention.

In step 52, a camera captures an image.

In step 54, the software receives a value for radial distortionparameter k from a user.

In step 56, the software corrects the radial exposure using equation 3.1with radial distortion parameter k and displays the corrected image tothe user.

In step 58, the software determines if the user approves the correctedimage. If so, step 58 is followed by step 60. Otherwise step 58 isfollowed by step 54 and the steps described above are repeated until.

In step 60, the software saves the corrected image as the result. FIG.4B shows an exemplary result.

Auto Picture Enhancement

FIG. 9 illustrates a method 70 for software to enhance an image in oneembodiment of the invention. The software can compensate for noise, poorcontrast, color cast, and poor focus.

In step 72, a camera captures an image.

The software then applies steps 74A, 74B, and 74C individually totransform the original image into three enhanced images.

In step 74A, the software performs an auto-level transformation to theoriginal image. The auto-level transformation extends the RGB channels'range to [0,255]. Assume that current range of channel C is [min^((C)),max^((C))], C∈{R,G,B}, the auto-level equation is: $\begin{matrix}{g_{new}^{(C)} = {\frac{255( {g^{(C)} - \min^{(C)}} )}{\max^{(C)}{- \min^{(C)}}}.}} & (4)\end{matrix}$In Equation 4, g_(new) ^((C)) is the new value of channel C, g^((C)) isthe original value of channel C, min^((C)) is a minimum value of channelC, and max^((C)) is a maximum value of the channel C. The transformcoefficients in the RGB channels are different and may cause the colorof the image to change after the transformation.

In step 74B, the software performs an auto-contrast transformation tothe original image. Assume that the range of RGB channels is [min^((C)),max^((C))], C∈{R,G,B}, min=MIN(min^((R)), min^((G)), min^((B))),max=MAX(max^((R)), max^((G)), max^((B)), the auto-contrast equation isdefined as: $\begin{matrix}{g_{new}^{(C)} = {\frac{255( {g^{(C)} - \min} )}{\max - \min}.}} & (5)\end{matrix}$In Equation 5, g_(new) ^((C)) is the new value of channel C and g^((C))is the original value of channel C.

In step 74C, the software performs an auto-brightness transformation tothe original image. While auto-level and auto-contrast transformationsare applied to the RGB channels, auto-brightness transformation isapplied to the YCrCb channels. Auto-brightness will only change the Ychannel while keeping the other two channels unchanged. Assuming thatthe range of the Y channel is [min Y, max Y], the auto-brightnessequation is defined as $\begin{matrix}{{Y_{new} = \frac{255( {Y - {\min\quad Y}} )}{{\max\quad Y} - {\min\quad Y}}},{{C\quad r_{new}} = {C\quad r}},{{C\quad b_{new}} = {C\quad{b.}}}} & (6)\end{matrix}$In Equation 6, Y_(new) is the new value of channel Y, Y is the originalvalue of channel Y, Cr_(new) is the new value of channel Cr, Cr is theoriginal value of channel Cr, Cb_(new) is the new value of channel Cb,and Cb is the original value of channel Cb.

The software then applies steps 76, 78, and 80 individually to the threeenhanced images from steps 74A, 74B, and 74C to generate three finalcandidates from which the user can select.

In step 76, the software performs an auto-color cast correction to eachof the enhanced images. Assuming color cast is a translation inchromatic space, then the equations for channels YCrCb are defined as:Cr _(new) =Cr _(old) +Δcr  (7.1)Cb _(new) =Cb _(old) +Δcb  (7.2)Y_(new)=Y_(old)  (7.3)In Equations 7.1, 7.2, and 7.3, Cr_(new) is the new value of channel Cr,Cr_(old) is the original value of channel Cr, Δcr is the color cast inchannel Cr, Cb_(new) is the new value of channel Cb, Cb_(old) is theoriginal value of channel Cb, Δcb is the color cast in channel Cb,Y_(new) is the new value of channel Y, and Y_(old) is the original valueof channel Y.

If color cast (Δcr,Δcb) is known, then the color cast can be removedfrom the image. To determine the color cast, the software first createsa 2D histogram Hist[Cr,Cb] and then determines the nearest peak from theorigin (0,0). Hist[Cr,Cb] is defined as the count of points (e.g.,pixels) with the chromatic value (Cr,Cb). A peak is located at aparticular (cr,cb) if (1) Hist(cr,cb)>Hist[cr−1,cb], (2)Hist(cr,cb)>Hist[cr+1,cb], (3) Hist(cr,cb)>Hist[cr,cb−1], and (4)Hist(cr,cb)>Hist[cr,cb+1]. After determining the nearest peak from theorigin, the software moves this peak to the origin. To do so, thesoftware sets (Δcr, Δcb)=(−cr,−cb) that is the nearest peak.

In step 78, the software performs an auto-gamma correction to each ofthe enhanced images. Gamma correction equation is defined as:$\begin{matrix}{Y_{new} = {255{( \frac{Y_{old}}{255} )^{\alpha}.}}} & (8)\end{matrix}$In Equation 8, Y_(new) is the new value of channel Y, Y_(old) is theoriginal value of channel Y, and α is the value that maximizes theentropy of image after correction. α can be estimated by maximizingEquation 9 below. $\begin{matrix}{\sum\limits_{1 \leq x \leq 256}^{\quad}\quad{{h(x)}{{\alpha( \frac{x}{256} )}^{\alpha - 1}.}}} & (9)\end{matrix}$In Equation 9, h(x) is the histogram of Y channel before correction andx is the gray level of Y channel. Equation 9 assumes that the gray levelof Y channel is from 1 to 256.

In step 80, the software performs a sharpness correction to each of theenhanced images. The sharpness correction uses a Laplacian filter as abasic operator but the filter is only applied to strong edges onlyinstead of the entire image. The sharpness filter is defined as:$\begin{matrix}{{I_{new}( {x,y} )} = \{ {\begin{matrix}{{I_{old}( {x,y} )},{{{if}\quad{{\nabla{I( {x,y} )}}}} \leq T}} \\{{{I_{old}( {x,y} )} + {\lambda{{\nabla^{2}{I( {x,y} )}}}}},{{{if}\quad{{\nabla{I( {x,y} )}}}} > T}}\end{matrix},} } & (10.1)\end{matrix}$where:|∇I(x,y)|=√{square root over (I _(x) ²(x,y)+I _(y) ²(x,y))},  (10.2)I _(x)(x,y)=I(x+1,y)−I(x,y),  (10.3)I _(y)(x,y)=I(x,y+1)−I(x,y), and   (10.4) $\begin{matrix}{{{\nabla^{2}{I( {x,y} )}}} = {{I( {x,y} )} - {\frac{1}{8}{\sum\limits_{i = {- 1}}^{1}\quad{\sum\limits_{j = {- 1}}^{1}\quad{{I( {{x + i},{y + j}} )}.}}}}}} & (10.5)\end{matrix}$In Equations 10.1, 10.2, 10.3, 10.4, and 10.5, T is a threshold valuethat determines if there is a strong edge, λ is a sharpness parameter,I( ) is intensity value at a point, I_(new)(x,y) is the new intensityvalue at point (x,y), and I_(old)(x,y) is the old intensity value at thepoint (x,y). In one embodiment, sharpness parameter λ=0.1.

In step 82, the software presents the three results to the user andprompts the user to select one of the three as the final result. FIGS.6B, 7B, and 8B illustrate exemplary results that can be selected by theuser. Specifically, FIG. 6B illustrates the result of the image in FIG.6A that has undergone auto-contrast, auto-color cast, auto-gammacorrection, and smart-sharpen modifications. FIG. 7B illustrates theresult of the image in FIG. 7A that has undergone auto-level, auto-colorcast, auto-gamma correction, and smart-sharpen modifications. FIG. 8Billustrates the result of the image in FIG. 8A that has undergoneauto-brightness, auto-color cast, auto-gamma correction, andsmart-sharpen modifications.

Various other adaptations and combinations of features of theembodiments disclosed are within the scope of the invention. Forexample, more than two local distortion parameters can be determinedfrom more than two lines and the global distortion parameter can beselected more these more than two local distortion parameters. Numerousembodiments are encompassed by the following claims.

1. A method for correcting a radial exposure distortion in an image,comprising: calculating new intensity values of points in the image withfollowing equations:I′(x,y)=I(x,y)^(1+k(x) ² ^(+y) ²⁾ , where I′(x,y) is an intensity valueof a point in the image with radial exposure, I(x,y) is an intensityvalue of the point without radial exposure, (x,y) are the coordinates ofthe point, and k is a radial distortion parameter.
 2. The method ofclaim 1, further comprising receiving the radial distortion parameterfrom a user.